Email Processing & Classification

1. Introduction

2. Project Structure

3. Core Components

4. Architecture Overview

5. Detailed Component Analysis

6. Dependency Analysis

7. Performance Considerations

8. Troubleshooting Guide

9. Conclusion

10. Appendices

This document explains the email processing and classification system that powers the notification pipeline for general notices and updates. It covers:

• Notice classification algorithms using LLM-based prompts

• Email parsing pipeline (headers, bodies, metadata)

• Notice types, priority, filtering, and content enrichment

• Integration with email clients, authentication, and batch processing

• Examples of processed data structures, accuracy considerations, and handling of malformed or suspicious emails

The email processing system centers around a LangGraph pipeline that classifies incoming emails and extracts structured notices. It integrates with:

• Google Groups client for fetching unread emails

• LLM prompts for classification and extraction

• Database persistence for notices and policy documents

• Notification dispatch for Telegram and web push

Diagram sources

• email_notice_service.py

• notice_formatter_service.py

• google_groups_client.py

• placement_policy_service.py

• database_service.py

• notification_service.py

Section sources

• README.md

• main.py

• EmailNoticeService: Orchestrates LLM-based classification and extraction for general notices; handles placement policy detection and fallbacks.

• NoticeFormatterService: Provides classification, enrichment, and formatting for notices and job postings.

• GoogleGroupsClient: Decodes email headers, extracts forwarded metadata, and normalizes dates to IST.

• PlacementPolicyService: Specialized extraction for placement policy updates into structured Markdown with TOC generation.

• DatabaseService: Persists notices, jobs, policies, and manages deduplication and stats.

• NotificationService: Routes formatted notices to Telegram and web push channels.

• Webhook server: Exposes health, stats, and notification endpoints for external integrations.

Section sources

• email_notice_service.py

• notice_formatter_service.py

• google_groups_client.py

• placement_policy_service.py

• database_service.py

• notification_service.py

• webhook_server.py

The email processing pipeline follows a LangGraph workflow:

• Input: Unread email IDs fetched from Google Groups

• Nodes: classify → extract_notice → validate → display_results

• Decision edges: classify → extract_notice if relevant; retry extraction on validation errors; skip if not a valid notice

• Outputs: Persisted NoticeDocument or policy updates

Diagram sources

• email_notice_service.py

• google_groups_client.py

• database_service.py

• placement_policy_service.py

EmailNoticeService: Classification and Extraction#

• Classification: Always marks as relevant and delegates to LLM for classification; rejects irrelevant/spam/placement-offer emails.

• Extraction: Uses a structured prompt to return JSON with fields for title, content, type, source, deadlines, links, and type-specific fields.

• Validation: Ensures minimum length for title/content and presence of type.

• Retry logic: Retries extraction up to two times on validation errors.

• Policy detection: Detects placement policy updates and triggers a secondary extraction with a specialized prompt.

Diagram sources

• email_notice_service.py

Section sources

• email_notice_service.py

• email_notice_service.py

• email_notice_service.py

NoticeFormatterService: Classification, Matching, and Formatting#

• Classification: Single-label classifier for categories: update, shortlisting, announcement, hackathon, webinar, job posting.

• Matching: Extracts company names and fuzzy-matches against job listings for enrichment.

• Formatting: Produces Telegram-ready messages with consistent structure, IST date formatting, and optional job enrichment.

Diagram sources

• notice_formatter_service.py

Section sources

• notice_formatter_service.py

• notice_formatter_service.py

• notice_formatter_service.py

• notice_formatter_service.py

GoogleGroupsClient: Email Parsing and Metadata Extraction#

• Fetches unread emails and parses multipart messages.

• Extracts forwarded sender and forwarded date, normalizing to IST.

• Provides safe marking/unmarking of emails as read/unread.

Diagram sources

• google_groups_client.py

Section sources

• google_groups_client.py

• google_groups_client.py

• google_groups_client.py

• google_groups_client.py

PlacementPolicyService: Policy Extraction and Storage#

• Detects placement policy emails and converts them into structured Markdown with TOC.

• Generates slugs and validates years from content.

• Upserts policy documents into MongoDB.

Diagram sources

• placement_policy_service.py

• placement_policy_service.py

Section sources

• placement_policy_service.py

• placement_policy_service.py

• placement_policy_service.py

• placement_policy_service.py

DatabaseService: Persistence and Deduplication#

• Saves notices and jobs, deduplicates by ID, and tracks sent status.

• Provides stats and retrieval helpers for unsent notices and collections.

Diagram sources

• database_service.py

Section sources

• database_service.py

• database_service.py

• database_service.py

• database_service.py

• database_service.py

NotificationService: Channel Routing and Delivery#

• Aggregates channels (Telegram, Web Push) and broadcasts messages to unsent notices.

• Supports targeted channels and detailed per-channel results.

Diagram sources

• notification_service.py

• notification_runner.py

Section sources

• notification_service.py

• notification_runner.py

Webhook Server: External Integrations#

• Exposes health, stats, push subscription, and notification endpoints.

• Enables external systems to trigger update jobs and send notifications.

Diagram sources

• webhook_server.py

Section sources

• webhook_server.py

• webhook_server.py

• webhook_server.py

• EmailNoticeService depends on GoogleGroupsClient for fetching, LLM for classification/extraction, and DatabaseService for persistence.

• NoticeFormatterService depends on Notice and Job models and uses LLM for classification and extraction.

• PlacementPolicyService depends on DatabaseService for upserting policy documents.

• NotificationService depends on channel implementations and DatabaseService for unsent notices.

• Webhook server composes NotificationService and WebPushService for integrations.

Diagram sources

• email_notice_service.py

• notice_formatter_service.py

• placement_policy_service.py

• notification_service.py

• webhook_server.py

Section sources

• email_notice_service.py

• notice_formatter_service.py

• placement_policy_service.py

• notification_service.py

• webhook_server.py

• Batch processing: The CLI orchestrator fetches unread IDs and processes emails sequentially, marking as read upon success to prevent reprocessing.

• Retry strategy: Up to two retries for extraction failures to improve robustness.

• Lazy enrichment: Jobs are enriched only when matched by the LLM, minimizing expensive API calls.

• Connection reuse: GoogleGroupsClient connects per-operation and disconnects to avoid stale connections.

• Logging and daemon mode: Centralized logging and daemon mode reduce overhead in production.

[No sources needed since this section provides general guidance]

Common issues and resolutions:

• Authentication failures (IMAP): Verify placement email and app password environment variables.

• LLM extraction errors: Inspect validation errors and retry attempts; ensure prompt compliance.

• Database connectivity: Confirm MongoDB connection string and collection initialization.

• Webhook endpoints: Check VAPID keys and CORS configuration for push endpoints.

• Duplicate notices: DatabaseService deduplicates by ID; ensure consistent notice IDs.

Section sources

• google_groups_client.py

• email_notice_service.py

• database_service.py

• webhook_server.py

• database_service.py

The email processing and classification system leverages LLM-driven prompts to reliably extract and structure notices from multiple sources. It integrates seamlessly with email clients, enforces deduplication, and delivers notifications across channels while maintaining a modular, testable architecture.

[No sources needed since this section summarizes without analyzing specific files]

Data Model: ExtractedNotice and NoticeDocument#

• ExtractedNotice: Fields include is_notice, rejection_reason, title, content, type, source, deadline, links, additional_info, and type-specific fields (students, company_name, role, package, location, eligibility_criteria, hiring_flow, job_type, event_name, topic, theme, speaker, date, time, registration_link, start_date, end_date, registration_deadline, prize_pool, team_size, organizer).

• NoticeDocument: Stored representation with id, title, content, author, type, source, formatted_message, createdAt, updatedAt, sent_to_telegram, time_sent, deadline, links, students, students_count.

Section sources

• email_notice_service.py

Classification Criteria and Notice Types#

• Notice types supported: announcement, hackathon, job_posting, shortlisting, update, webinar, reminder, internship_noc.

• Classification is LLM-based; irrelevant/spam/placement-offer emails are rejected early.

• Category classification for formatting: update, shortlisting, announcement, hackathon, webinar, job posting.

Section sources

• email_notice_service.py

• notice_formatter_service.py

Priority Assignment and Filtering#

• Priority is implicit: placement offers are handled by a separate pipeline; general notices are processed after placement offers in the orchestrated email update flow.

• Filtering: LLM determines relevance; validation ensures minimal quality; deduplication prevents repeated notifications.

Section sources

• main.py

• email_notice_service.py

• database_service.py

Integration with Email Clients and Authentication#

• GoogleGroupsClient authenticates via IMAP and app password; supports fetching unread emails, extracting forwarded metadata, and marking read/unread.

• Configuration: Environment variables for placement email and app password.

Section sources

• google_groups_client.py

• config.py

Batch Processing Workflows#

• CLI orchestrator: Iterates unread emails, tries placement offer detection first, then general notice extraction, persists results, and marks as read.

• NotificationRunner: Sends unsent notices via Telegram and/or Web Push channels.

Section sources

• main.py

• notification_runner.py

Spam Detection and Duplicate Filtering#

• Spam detection: LLM prompt explicitly rejects spam or irrelevant content.

• Duplicate filtering: DatabaseService checks existence by ID before insertion.

Section sources

• email_notice_service.py

• database_service.py

Content Enrichment and Formatting#

• NoticeFormatterService enriches notices with job matching and formats Telegram-ready messages with IST timestamps and attribution.

• Email date normalization: Forwarded dates and email Date headers are normalized to IST.

Section sources

• notice_formatter_service.py

• google_groups_client.py